Image capturing apparatus and control method thereof

ABSTRACT

An image capturing apparatus reads out a signal from pixels of the image sensor, sets a region in which a plurality of signals having different viewpoints are read out from each pixel of an image sensor, acquires first depth information for detecting an object using a signal that has been read out from a first region, acquires second depth information for detecting a focus state of the object using a signal that has been read out from a second region, and variably controls a ratio of screens in which the first region is set and a ratio of screens in which the second region is set.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image capturing apparatus thatperforms focus adjustment and object detection.

Description of the Related Art

Conventionally, techniques for performing focus detection based on thephase difference of image signals acquired by an image sensor that haspupil-divided pixels using a microlens are known (Japanese PatentLaid-Open No. 2007-325139). In Japanese Patent Laid-Open No.2007-325139, each pupil-divided pixel receives, via the microlens, lightbeams that have passed through different pupil regions of an imagingoptical system. Also, image signals can be acquired by adding togetherimage signals.

In focus detection by a phase difference method as described above,determining the amount of image signals that are to be read out forfocus adjustment and subjected to calculation processing is a veryimportant factor in terms of the detection accuracy and the processingspeed. In addition, in the case of an image sensor in which each pixelis divided into two, if all the image signals are taken in, the dataamount will be twice the data amount of data for a captured image,placing a large load on later-stage processing circuits.

In view of this, image capturing apparatuses have been proposed in whicha distance information acquisition region for focus adjustment can besuitably set in the image sensor, and the time for reading out imagesignals from the image sensor is reduced (Japanese Patent Laid-Open No.2012-155095). Also, image capturing apparatuses that can generate thedistribution of the distances (a distance map) of objects in an imageusing image signals acquired from a distance information acquisitionregion for focus adjustment have been proposed (Japanese PatentLaid-Open No. 2014-074891). By using the distance map of Japanese PatentLaid-Open No. 2014-074891, distance information of a main object andanother object in the image is acquired, and the main object can bedetected in cases such as where the main object and the other objectpass each other.

However, in the above-described conventional techniques, the distanceinformation acquisition region for focus adjustment and the distanceinformation acquisition region for object detection do not necessarilymatch, and thus there is a possibility that, if one of the regions isfocused on, accuracy for the other region deteriorates.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of theaforementioned problems, and makes it possible to ensure both focusadjustment accuracy and object detection accuracy.

In order to solve the aforementioned problems, the present inventionprovides an image capturing apparatus comprising: an image sensor; areadout unit configured to read out a signal from pixels of the imagesensor; a setting unit configured to set a region in which a pluralityof signals having different viewpoints are read out from each pixel ofthe image sensor by the readout unit; a first information acquisitionunit configured to acquire first depth information for detecting anobject using a signal that has been read out from a first region set bythe setting unit; a second information acquisition unit configured toacquire second depth information for detecting a focus state of theobject using a signal that has been read out from a second region set bythe setting unit; and a control unit configured to variably control aratio of screens in which the first region is set by the setting unitand a ratio of screens in which the second region is set.

In order to solve the aforementioned problems, the present inventionprovides an image capturing apparatus comprising: an image sensor; areadout unit configured to read out a signal from pixels of the imagesensor; a setting unit configured to set a region in which a pluralityof signals having different viewpoints are read out from each pixel ofthe image sensor by the readout unit; a first information acquisitionunit configured to acquire first depth information for detecting anobject using a signal that has been read out from a first region set bythe setting unit; a second information acquisition unit configured toacquire second depth information for detecting a focus state of theobject using a signal that has been read out from a second region set bythe setting unit; and a control unit configured to variably controlratios of an entire screen occupied by the first region and the secondregion set by the setting unit.

In order to solve the aforementioned problems, the present inventionprovides a control method of an image capturing apparatus which has animage sensor, a readout unit configured to read out a signal from pixelsof the image sensor, and a setting unit configured to set a region inwhich a plurality of signals having different viewpoints are read outfrom each pixel of the image sensor by the readout unit, the methodcomprising: acquiring first depth information for detecting an objectusing a signal that has been read out from a first region set by thesetting unit; acquiring second depth information for detecting a focusstate of the object using a signal that has been read out from a secondregion set by the setting unit; and variably controlling a ratio ofscreens in which the first region is set by the setting unit and a ratioof screens in which the second region is set.

In order to solve the aforementioned problems, the present inventionprovides a control method of an image capturing apparatus which has animage sensor, a readout unit configured to read out a signal from pixelsof the image sensor, and a setting unit configured to set a region inwhich a plurality of signals having different viewpoints are read outfrom each pixel of the image sensor by the readout unit, the methodcomprising: acquiring first depth information for detecting an objectusing a signal that has been read out from a first region set by thesetting unit; acquiring second depth information for detecting a focusstate of the object using a signal that has been read out from a secondregion set by the setting unit; and variably controlling ratios of anentire screen occupied by the first region and the second region set bythe setting unit.

In order to solve the aforementioned problems, the present inventionprovides a computer-readable storage medium storing a program forcausing a computer to execute a control method of an image capturingapparatus which has an image sensor, a readout unit configured to readout a signal from pixels of the image sensor, and a setting unitconfigured to set a region in which a plurality of signals havingdifferent viewpoints are read out from each pixel of the image sensor bythe readout unit, the method comprising: acquiring first depthinformation for detecting an object using a signal that has been readout from a first region set by the setting unit; acquiring second depthinformation for detecting a focus state of the object using a signalthat has been read out from a second region set by the setting unit; andvariably controlling a ratio of screens in which the first region is setby the setting unit and a ratio of screens in which the second region isset.

In order to solve the aforementioned problems, the present inventionprovides a computer-readable storage medium storing a program forcausing a computer to execute a control method of an image capturingapparatus which has an image sensor, a readout unit configured to readout a signal from pixels of the image sensor, and a setting unitconfigured to set a region in which a plurality of signals havingdifferent viewpoints are read out from each pixel of the image sensor bythe readout unit, the method comprising: acquiring first depthinformation for detecting an object using a signal that has been readout from a first region set by the setting unit; acquiring second depthinformation for detecting a focus state of the object using a signalthat has been read out from a second region set by the setting unit; andvariably controlling ratios of an entire screen occupied by the firstregion and the second region set by the setting unit.

According to the present invention, both focus adjustment accuracy andobject detection accuracy can be ensured.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the configuration of an imagecapturing apparatus of embodiments of the present invention.

FIG. 2 is a diagram schematically showing the pixel arrangement of animage sensor of the embodiments.

FIG. 3 is a diagram schematically showing the relationship between lightbeams coming from an exit pupil of a photographing lens and a pixel.

FIG. 4 is a configuration diagram of the image sensor of theembodiments.

FIG. 5A is a diagram showing the circuit configuration of a unit pixelof the image sensor of the embodiments.

FIG. 5B is a configuration diagram of a readout circuit for a column ofunit pixels of the image sensor of the embodiments.

FIG. 5C is a diagram showing a focus detection frame that is set for thepixel array of the image sensor of the embodiments.

FIGS. 6A to 6C are timing charts of an operation of reading out a row ofunit pixels of the image sensor of the embodiments.

FIG. 7 is a block diagram showing the configuration of an AF controlunit of a first embodiment.

FIG. 8 is a diagram illustrating a weight coefficient when calculatingoptical system driving information of the first embodiment.

FIG. 9 is a flowchart showing processing for setting a distanceinformation acquisition region performed by a distance informationacquisition region setting unit of the first embodiment.

FIGS. 10A to 10C are diagrams illustrating distance informationacquisition regions that are set by the region setting unit of the firstembodiment.

FIG. 11 is a diagram showing the configuration of an AF control unit ofa second embodiment.

FIG. 12 is a flowchart showing processing for setting a distanceinformation acquisition region performed by a distance informationacquisition region setting unit of the second embodiment.

FIG. 13 is a flowchart showing processing for setting a distanceinformation acquisition region performed by a distance informationacquisition region setting unit of a third embodiment.

FIGS. 14A to 14C are diagrams illustrating distance informationacquisition regions that are set by the region setting unit of the thirdembodiment.

FIG. 15 is a flowchart showing processing for setting a distanceinformation acquisition region performed by a distance informationacquisition region setting unit of a fourth embodiment.

FIGS. 16A to 16D are diagrams illustrating the background of theembodiments.

DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention will be described in detail below.The following embodiments are merely examples for practicing the presentinvention. The embodiments should be properly modified or changeddepending on various conditions and the structure of an apparatus towhich the present invention is applied. The present invention should notbe limited to the following embodiments. Also, parts of the embodimentsto be described later may be properly combined.

Description of Background

First, the background of embodiments of the present invention will bespecifically described with reference to FIGS. 16A to 16C.

FIGS. 16A to 16C illustrate the relationship between a distanceinformation acquisition region (for AF control) and a distance mapthereof that are required for focus adjustment and distance informationacquisition regions (for main object tracking) and a distance mapthereof that are required for object detection, and those distanceinformation acquisition regions and distance maps are suitably set onthe imaging screen at the time of AF control during image shooting. Notethat in FIGS. 16A to 16C, (a-1) to (a-4), (b-1) to (b-4), (c-1) to (c-4)and (d-1) to (d-4) indicate the frames of captured image signals in timeseries.

FIG. 16A illustrates image signals in the case where a distanceinformation acquisition region for AF control is set on an imagingscreen. In FIG. 16A, (a-1) indicates image signals and a distanceinformation acquisition region for AF control at a certain time,reference numeral 1600 denotes a main object, reference numeral 1601denotes an object other than the main object, and reference numeral 1602denotes a distance information acquisition region. It suffices thatdistance information of the main object can be acquired for AF control,and thus it suffices for the distance information acquisition regionthat is set for AF control to encompass the main object. Therefore, thedistance information acquisition region 1602 is locally set on thescreen as in (a-1). In addition, as the time elapses as indicated by(a-1) to (a-4), the object 1601 other than the main object approachesthe main object 1600 (a-2), the object 1601 other than the main objectoverlaps the main object 1600 (a-3), and then the object 1601 other thanthe main object disappears from the screen, and only the main object1600 remains (a-4). In other words, the images (a-1) to (a-4) indicate ascene in which the main object and the other object pass each other.

FIG. 16B illustrates distance maps acquired from the distanceinformation acquisition region 1602 in FIG. 16A. In FIG. 16B, (b-1) is adistance map acquired from the distance information acquisition regionof (a-1), where a crosshatched region 1603 indicates the distance of themain object 1600, and a solid white portion 1604 indicates the distanceof the background. In addition, a solid black portion 1605 is a regionoutside of the distance information acquisition region, and thusdistance information cannot be acquired. Moreover, a hatched portion1606 of (b-3) indicates the distance of the object 1601 other than themain object in (a-3).

In the distance map in FIG. 16B, the distance information acquisitionregion for AF control is set for the main object 1600, and thus AFcontrol accuracy does not deteriorate. On the other hand, in the case ofdetecting the main object 1600 using the distance map in FIG. 16B, onlythe distance 1603 of the main object 1600 can be acquired in (b-1) and(b-2), and it appears as if the distance of the main object 1600 changedsuddenly in (b-3). In actuality, the object 1601 other than the mainobject overlapped the main object 1600 and thus the distance changed,but there is a possibility that correct determination cannot beperformed.

FIG. 16C illustrates image signals in the case where distanceinformation acquisition regions for object detection are set on theimaging screen. In FIG. 16C, (c-1) indicates image signals and distanceinformation acquisition regions for object detection at a certain time.The state of the surroundings of the main object needs to be determinedin order to detect the main object 1600 using distance information.Therefore, a distance information acquisition region needs to be setsuch that the entire screen can be viewed. Therefore, distanceinformation acquisition regions 1607 are discretely set over the entirescreen as in (c-1). Note that the state where time elapses as in (c-1)to (c-4) and the state where the object 1601 other than the main objectoverlaps the main object 1600 in (c-3) are similar to (a-3) in FIG. 16A.

FIG. 16D illustrates a distance map acquired from the distanceinformation acquisition regions 1607 in FIG. 16C. In FIG. 16D, thedistance information acquisition regions 1607 are set such that theentire screen can be viewed, and thus it can be realized from thedistance map that the object 1601 other than the main object approachesthe main object 1600 as the time elapses from (d-1) to (d-2). Inaddition, in (d-3), the main object 1600 and the object 1601 other thanthe main object overlap each other, but since the object 1601 other thanthe main object approaches the main object 1600 from (d-1) to (d-2), andthe object 1601 has a distance 1609 indicated by hatched lines, it canbe determined that there is a high possibility that the main object 1600and the object 1601 other than the main object overlap each other. Onthe other hand, a distance 1608 of the main object 1600 can only beacquired discretely, and thus accuracy of AF control deteriorates.

In this manner, the distance information acquisition region for AFcontrol does not necessarily match the distance information acquisitionregions for object detection (main object tracking), and thus if one ofthe regions is focused on, there is a possibility that accuracy for theother region deteriorates.

In view of this, in the following embodiments, both AF control accuracyand object detection accuracy can be ensured by variably controllingratios for the distance information acquisition region for AF controland the distance information acquisition regions for object detection inunits of frames or within a frame at the time of AF control during imageshooting.

First Embodiment

In this embodiment, an example will be described in which an imagecapturing apparatus is realized by a digital video camera that has anautofocus (AF) function by a contrast detection method and a phasedifference detection method, and also an object detection function (amain object tracking function), but the present invention can be appliedto electronic devices such as smart phones, which are one type of amobile phone, and tablet terminals.

Apparatus Configuration

The configuration of an image capturing apparatus 100 of this embodimentwill be described below with reference to FIG. 1.

In FIG. 1, an optical system 1 includes a zoom lens, a focus lens and adiaphragm. An optical system driving unit 2 controls the optical system1 based on optical system drive information that is output from an AFcontrol unit 8 to be described later. An image sensor 3 is provided withphotoelectric conversion elements of a CMOS or the like, and converts anobject image formed on a light receiving surface by the optical system 1into electrical signals, and outputs image signals.

The image sensor driving unit 4 drives the image sensor 3 based oninformation regarding an object distance information acquisition region(hereinafter, region information) from the AF control unit 8 so as tocontrol readout of the image signals. Note that the image sensor 3 ofthis embodiment has a plurality of pixel portions for receiving luminousbeams that have passed through different pupil regions of the opticalsystem 1, and outputting pupil-divided image signals. Also, the imagesignals (A image signals and B image signals) having different parallax(or viewpoints) can be individually read out from pupil-divided pixelsby a drive pulse from the image sensor driving unit 4. The circuitconfiguration of the image sensor 3 will be described later withreference to FIG. 2 to FIG. 6C.

A contrast evaluation value calculation unit 5 calculates a contrastevaluation value (evaluation information) based on the image signalsfrom the image sensor 3 and focus detection frame information from theAF control unit 8, and outputs the contrast evaluation value to the AFcontrol unit 8.

A focus detection unit 6 calculates distance information for AF control(second depth information) and distance information for object detectionwithin the screen (first depth information and distance map data) basedon the image signals from the image sensor 3 and region information fromthe AF control unit 8, and outputs the distance information for AFcontrol and the distance information for object detection to an objecttracking unit 7 and the AF control unit 8.

The object tracking unit 7 detects objects in a captured image based onthe image signals from the image sensor 3 and the distance informationfrom the AF control unit 8, identifies a main object among the detectedobjects, and outputs, to the AF control unit 8, information regardingthe position and the size of the main object (hereinafter, objectinformation).

Under control of a system control unit 13, the AF control unit 8 outputsfocus detection frame information and region information to the contrastevaluation value calculation unit 5, and outputs the region informationto the focus detection unit 6. The AF control unit 8 also acquires acontrast evaluation value from the contrast evaluation value calculationunit 5 and distance information from the focus detection unit 6, andoutputs a control signal to the optical system driving unit 2 and theimage sensor driving unit 4.

A signal processing unit 9 generates image signals acquired by addingtogether image signals from the image sensor 3, performs predeterminedsignal processing, and outputs image signals for display or forrecording. The signal processing unit 9 also performs image processingsuch as color conversion, white balance correction and gamma correction,resolution conversion processing, image compression conversion and thelike on the generated image signals, and outputs, to a recording unit 10and a display unit 11, image signals for display or for recording.

The recording unit 10 is a memory card, a hard disk or the like in whichthe image signals generated by the signal processing unit 9 arerecorded, and from which images that have been already recorded are readout. The display unit 11 is a liquid crystal panel (LCD) or the likethat displays images generated by the signal processing unit 9, variousmenu screens and the like. An operation unit 12 is constituted byvarious switches (e.g., AF on/off and zoom) for receiving a useroperation, and transmits instructions from the user to the systemcontrol unit 13.

The system control unit 13 includes a CPU, a RAM and a ROM forintegrally controlling various functions of the image capturingapparatus 100, a dedicated circuit and the like. The CPU executes acontrol sequence, which will be described later, by loading, to the RAMserving as a work memory, a program stored in the ROM that is anon-volatile memory, and executing the program.

Configuration of Image Sensor

FIG. 2 is a schematic diagram showing the pixel arrangement of the imagesensor 3. Unit pixels 200 are arranged in a matrix, and R (Red)/G(Green)/B (Blue) color filters are arranged on the unit pixels 200 in aBayer pattern. In addition, subpixels a and b are arranged in each ofthe unit pixels 200, and photodiodes (hereinafter, PDs) 201 a and 201 bare respectively arranged in the subpixels a and b. Imaging signals thatare output from the subpixels a and b are used for focus detection, andan a/b composite signal that is a signal acquired by adding the imagingsignals output from the subpixel a and the subpixel b is used for imagegeneration.

FIG. 3 shows the relationship between light beams coming from differentregions of the exit pupil of the optical system 1 and the unit pixel200, and the same reference numerals are assigned to constituentelements similar to those in FIG. 2.

As shown in FIG. 3, a color filter 301 and a microlens 302 are formed oneach of the unit pixels 200. Specifically, the PDs 201 a and 201 b inthe unit pixel 200 are assigned to one microlens 302. Light that haspassed through an exit pupil 303 of the lens enters the unit pixel 200centered on an optical axis 304. A light beam that passes through apupil region 305, which is a region constituting a portion of the exitpupil 303, passes through the microlens 302, and is received by thesubpixel a. On the other hand, a light beam that passes through a pupilregion 306, which is a region constituting another portion of the exitpupil 303, passes through the microlens 302, and is received by thesubpixel b. Therefore, the subpixels a and b respectively receive lightfrom the separate pupil regions 305 and 306 of the exit pupil 303 of theoptical system 1. Therefore, phase difference (imaging plane) focusdetection is made possible by comparing an output signal of the subpixela (A image signal) with an output signal of the subpixel b (B imagesignal) that have different parallax (or viewpoints) as described above.

FIG. 4 shows the circuit configuration of the image sensor 3. In a pixelregion PA, the unit pixels 200 are arranged in a matrix (n rows×kcolumns) as indicated by p11 to pkn. Here, the configuration of the unitpixel 200 will be described with reference to FIG. 5A. FIG. 5A is adiagram showing the circuit configuration of a unit pixel of the imagesensor.

In FIG. 5A, optical signals that have entered PDs (photoelectricconversion portion) 501 a and 501 b of the above-described subpixels aand b undergo photoelectric conversion performed by the PDs 501 a and501 b, and electric charges that correspond to an exposure amount areaccumulated in the PDs 501 a and 501 b. The electric charges accumulatedin the PDs 501 a and 501 b are transferred to an FD (floating diffusion)portion 503 (electric charge transfer) by raising signals txa and txbthat are respectively applied to the gates of transfer gates 502 a and502 b to the high level. The FD portion 503 is connected to the gate ofa floating diffusion amplifier 504 (hereinafter, expressed as an FDamplifier), and the amount of electric charges transferred from the PDs501 a and 501 b is converted into a voltage amount by the FD amplifier504.

The FD portion 503 is reset by raising, to the high level, a signal resthat is applied to the gate of an FD reset switch 505 for resetting theFD portion 503. In addition, in the case of resetting the electriccharges of the PDs 501 a and 501 b, the signal res as well as thesignals txa and txb are raised to the high level at the same time. Thisturns on both the transfer gates 502 a and 502 b and the FD reset switch505, and the PDs 501 a and 501 b are reset via the FD portion 503. Apixel signal that has been converted into a voltage by the FD amplifier504 is output to an output vout of the unit pixel 200 by raising asignal sel that is applied to the gate of a pixel selection switch 506to the high level.

As shown in FIG. 4, a vertical scanning circuit 401 supplies, to each ofthe unit pixels 200, driving signals such as res, txa, txb and sel forcontrolling the above-described switches of the unit pixel 200. Thesedriving signals res, txa, txb and sel are common to each row. Theoutputs vout of the unit pixels 200 are connected to a column commonreadout circuit 403 via a vertical output line 402 for each column.

Here, the configuration of the column common readout circuit 403 will bedescribed with reference to FIG. 5B.

The vertical output line 402 is provided for each column of unit pixels200, and is connected to the outputs vout of the unit pixels 200 for onecolumn. A current source 404 is connected to the vertical output line402, and a source follower circuit is constituted by the current source404 and the FD amplifiers 504 of the unit pixels 200 connected to thevertical output line 402.

In FIG. 5B, a clamp capacitor 601 has a capacity of C1, a feedbackcapacitor 602 has a capacity of C2, and an operational amplifier 603 hasa non-inverted input terminal connected to a reference power supplyVref. A switch 604 is used for causing two ends of the feedbackcapacitor 602 to short-circuit, and the switch 604 is controlled by asignal cfs.

Transfer switches 605 to 608 are used for respectively transferring, tosignal storage capacitors 609 to 612, signals read out from the unitpixels 200. The first S signal storage capacitor 609 stores a pixelsignal Sa that is output from the subpixel a by a readout operation tobe described later. Also, the second S signal storage capacitor 611stores an a/b composite signal Sab that is a signal acquired bycompositing (adding) a signal output from the subpixel a and a signaloutput from the subpixel b. Moreover, the first N signal storagecapacitor 610 and the second N signal storage capacitor 612 respectivelystore noise signals N of the unit pixels 200. The signal storagecapacitors 609 to 612 are respectively connected to outputs vsa, vna,vsb and vnb of the column common readout circuit 403.

Horizontal transfer switches 405 and 406 are respectively connected tothe outputs vsa and vna of the column common readout circuit 403. Thehorizontal transfer switches 405 and 406 are controlled by an outputsignal ha* (* is a column number) of a horizontal scanning circuit 411.

Also, horizontal transfer switches 407 and 408 are respectivelyconnected to the outputs vsb and vnb of the column common readoutcircuit 403. The horizontal transfer switches 407 and 408 are controlledby an output signal hb* (* is a column number) of the horizontalscanning circuit 411. Horizontal output lines 409 and 410 are connectedto an input of a differential amplifier 414, and the differentialamplifier 414 takes the difference between an S signal and an N signal,applies a predetermined gain at the same time, and outputs a finaloutput signal to an output terminal 415.

When a signal chres applied to the gates of horizontal output line resetswitches 412 and 413 is raised to the high level, the horizontal outputline reset switches 412 and 413 are turned on, and the horizontal outputlines 409 and 410 are reset to a reset voltage Vchres.

An operation of reading out A image signals and an operation of readingout A+B image signals that are composite signals of the A image signalsand B image signals will be described below.

FIG. 5C shows the relationship between distance information acquisitionregions for focus adjustment and distance information acquisitionregions for object detection that are set in the pixel region PA of theimage sensor 3. Focus detection frames 620 are set by the focusdetection unit 6, using region information from the AF control unit 8.

In the pixel region PA constituted by pixels of k columns×n rows,regions indicated by dotted lines are the focus detection frames 620. Aimage signals and A+B image signals are read out from unit pixel rows(pixel lines) included in distance information acquisition regions R1indicated by hatched portions, and are used for image generation, focusdetection and object detection. Only addition signals of A image signalsand B image signals are read out from unit pixel rows (pixel lines)included in regions R2 other than the distance information acquisitionregions R1, and are used only for image generation.

Note that as shown in FIG. 5C, if a plurality of regions R1 are set inthe vertical direction of the pixel region, the number of rows of theunit pixels 200 may be set differently in each of the regions R1.

Next, a readout operation of the image sensor 3 will be described withreference to FIG. 6A. FIG. 6A is a timing chart of the above-describedreadout operation performed on each row of the above-described regionsR2.

First, the operational amplifier 603 is brought into a buffer state byraising the signal cfs to the high level and turning on the switch 604.Next, the signal sel is raised to the high level, and the pixelselection switch 506 of a unit pixel is turned on. After that, thesignal res is lowered to the low level, and the FD reset switch 505 isturned off so as to release the resetting of the FD portion 503.

Subsequently, after the signal cfs is returned to the low level and theswitch 604 is turned off, signals tna and tnb are raised to the highlevel, and noise signals N are stored in the first N signal storagecapacitor 610 and the second N signal storage capacitor 612 via transferswitches 606 and 608.

Next, the signals tna and tnb are lowered to the low level, and thetransfer switches 606 and 608 are turned off. After that, a transferswitch 607 is turned on by raising a signal tsb to the high level, andthe transfer gates 502 a and 502 b are turned on by raising the signalstxa and txb to the high level. By this operation, signals acquired bycompositing electric charge signals accumulated in the PDs 501 a of thesubpixels a and electric charge signals accumulated in the PDs 501 b ofthe subpixels b are output to the vertical output line 402 via the FDamplifier 504 and the pixel selection switch 506. Signals of thevertical output line 402 are amplified by the operational amplifier 603using a gain that corresponds to the capacity ratio of the capacity C1of the clamp capacitor 601 to the capacity C2 of the feedback capacitor602, and are stored in the second S signal storage capacitor 611 via thetransfer switch 607 (the a/b composite signal Sab). After sequentiallyturning off the transfer gates 502 a and 502 b and the transfer switch607, the signal res is raised to the high level so as to turn on the FDreset switch 505, and the FD portion 503 is reset.

Next, the horizontal transfer switches 407 and 408 are turned on due toan output hb1 of the horizontal scanning circuit 411 rising to the highlevel. Accordingly, signals of the second S signal storage capacitor 611and the second N signal storage capacitor 612 are output to the outputterminal 415 via the horizontal output lines 409 and 410 and thedifferential amplifier 414. The horizontal scanning circuit 411 outputsthe a/b composite signals (the A+B image signals) for one row bysequentially raising selection signals hb1, hb2, . . . , hbk of eachcolumn to the high level. Note that while signals of each column areread out by the signals hb1 to hbk, the horizontal output line resetswitches 412 and 413 are turned on by raising the signal chres to thehigh level, and the horizontal output lines 409 and 410 are reset to thelevel of the reset voltage Vchres once.

The above-described operation is an operation of reading out each row ofunit pixels in the region R2. The A+B image signals are read out by thisoperation.

Subsequently, an operation of reading out each row of the regions R1will be described with reference to FIGS. 6B and 6C. FIG. 6B is a timingchart of an operations for readout of the A image signals. The operationof first raising the signal cfs to the high level, lowering the signalstna and tnb to the low level, and storing N signals in the first Nsignal storage capacitor 610 and the second N signal storage capacitor612 is similar to the operation described with reference to FIG. 6A.

When storing of the noise signals N ends, the transfer switch 605 isturned on by raising a signal tsa to the high level, and the transfergate 502 a is turned on by raising the signal txa to the high level.Signals accumulated in the PDs 501 a of the subpixels a are output tothe vertical output line 402 via the FD amplifier 504 and the pixelselection switch 506 by performing such an operation. Signals of thevertical output line 402 are amplified by the operational amplifier 603using a gain that corresponds to the capacity ratio of the capacity C1of the clamp capacitor 601 to the capacity C2 of the feedback capacitor602, and are stored in the first S signal storage capacitor 609 via thetransfer switch 605 (the pixel signal Sa).

Next, the horizontal transfer switches 405 and 406 are turned on due toan output ha1 of the horizontal scanning circuit 411 rising to the highlevel. Accordingly, signals of the first S signal storage capacitor 609and the first N signal storage capacitor 610 are output to the outputterminal 415 via the horizontal output lines 409 and 410 and thedifferential amplifier 414. The horizontal scanning circuit 411 outputsthe signals of the subpixels a (the A image signals) for one row bysequentially raising selection signals ha1, ha2, . . . , hak for eachcolumn to the high level.

Readout of the A image signals ends while the signal res remains at thelow level, and the signal sel remains at the high level. Accordingly,the A image signals on the FD portions 503 are held without being reset.

When readout of the A image signals ends, the procedure subsequentlytransitions to an operation of reading out the A+B image signals shownin FIG. 6C. The transfer switch 607 is turned on by raising the signaltsb to the high level, and the transfer gates 502 a and 502 b are turnedon by raising the signals txa and txb to the high level. Due to such anoperation, signals accumulated in the PDs 501 b of the subpixels b areadded to the signals of the subpixels a stored in the FD portion 503,and the added signals are output to the vertical output line 402 via theFD amplifier 504 and the pixel selection switch 506. The rest of theoperation is the same as the operation regarding the region R2 describedwith reference to FIG. 6A.

In such a manner, an operation of reading out each row in the regions R1ends. Accordingly, in the region R1, readout of the A image signals andreadout of the A+B image signals are performed, and the A image signalsand the A+B image signals are sequentially read out.

Shooting Operation

Next, an operation during image shooting performed by the imagecapturing apparatus 100 that has the above-described configuration willbe described.

First, the optical system 1 uses a driving signal from the opticalsystem driving unit 2 to drive the diaphragm and the lens, so as to forman object image whose brightness is set to be appropriate, on the lightreceiving surface of the image sensor 3. The image sensor 3 is driven bya drive pulse from the signal readout control unit 4, converts theobject image into electrical signals by photoelectric conversion, andoutputs the electrical signals as image signals.

Using a drive pulse that corresponds to region information from the AFcontrol unit 8, the image sensor driving unit 4 reads out the A imagesignals and reads out the A+B image signals from the region R1, andreads out the A+B image signals from the region R2 by theabove-described readout operation. The processing load is reduced byreading out the A image signals from a portion of the region in thismanner. Furthermore, in the region R1 from which the A image signalshave been read out, the AF control unit 8 acquires B image signals bysubtracting the A image signals from the A+B image signals, and performsAF control using the A image signals and the B image signals. Note thatAF control may be performed by individually reading out the A imagesignals and the B image signals from the region R1, and reading out theA+B image signals from the region R2 other than the region R1.

The contrast evaluation value calculation unit 5 calculates a contrastevaluation value in a focus detection frame based on image signals fromthe image sensor 3 and focus detection frame information from the AFcontrol unit 8, and outputs the contrast evaluation value to the AFcontrol unit 8. In this case, the contrast evaluation value calculationunit 5 adds the A image signals and the B image signals based on regioninformation from the AF control unit 8, applies the same format as theA+B image signals read out from regions R2 other than the distanceinformation acquisition regions R1, and calculates the contrastevaluation value.

Here, an overview of contrast AF will be described. The contrastevaluation value calculation unit 5 shifts a first focus detectionsignal acquired from the A image signal and a second focus detectionsignal acquired from the B image signal relatively in the pupil divisiondirection, adds those signals to generate a shift addition signal, andcalculates a contrast evaluation value from the generated shift additionsignal.

Letting a k-th first focus detection signal be A(k), a k-th second focusdetection signal be B(k), the range of the number k for the distanceinformation acquisition region R1 be W, the shift amount due to shiftprocessing be s1, and the shift range of the shift amount s1 be τ1, acontrast evaluation value RFCON is calculated using the followingexpression.

${{{RFCON}\left( {s\; 1} \right)} = {\max\limits_{k \in W}\left| {{A(k)} - {B\left( {k - {s\; 1}} \right)}} \right|}},{{s\; 1} \in {\tau 1}}$

Due to shift processing by the shift amount s, the k-th first focusdetection signal A (k) and a (k−s1)th second focus detection signal B(k−s1) are added in association with each other so as to generate ashift addition signal, and the contrast evaluation value RFCON (s1) iscalculated from the shift addition signal.

The focus detection unit 6 calculates distance information of the object(first depth information and second depth information) based on regioninformation from the AF control unit 8, using the A image signals readout from a distance information acquisition region for AF and a distanceinformation acquisition region for object detection that are variablycontrolled by the image sensor driving unit 4, and the B image signalacquired by subtracting the A image signal from the A+B image signal.Note that in this embodiment, distance information is phase differenceinformation (defocus amount) for performing phase difference (imagingplane) AF.

Here, an overview of phase difference AF will be described. The focusdetection unit 6 shifts a first focus detection signal acquired from theA image signal and a second focus detection signal acquired from the Bimage signal relatively in the pupil division direction, and calculatesa correlation amount indicating a signal matching degree. Letting a k-thfirst focus detection signal be A(k), a k-th second focus detectionsignal be B(k), the range of the number k for the distance informationacquisition region R1 be W, the shift amount due to shift processing bes2, and the shift range of the shift amount s2 be τ2, a correlationamount COR is calculated using the following expression.

${{{COR}\left( {s\; 2} \right)} = {\sum\limits_{k \in W}\left| {{A(k)} - {B\left( {k - {s\; 2}} \right)}} \right|}},{{s\; 2} \in {\tau 2}}$

Due to shift processing by the shift amount s2, the k-th first focusdetection signal A (k) and a (k−s2)th second focus detection signal B(k−s2) are associated with each other, and subtraction is performed togenerate a shift subtraction signal, and the sum of the k signals isobtained within the range W corresponding to the distance informationacquisition region so as to calculate the correlation amount COR (s2).After that, the shift amount of a real value at which the correlationamount is a minimum value is calculated from the correlation amount byperforming subpixel calculation, and is denoted by an image shift amountp1. The image shift amount p1 is multiplied by the image height of thefocus detection region, the F-number of the imaging lens (imagingoptical system) and a first conversion coefficient K1 that correspondsto the exit pupil distance so as to detect the defocus amount.

Note that in this embodiment, an example is described in which the focusdetection unit 6 calculates distance information from A image signalsand B image signals having different parallax (or viewpoints), but“information corresponding to depth” that is not converted into“distance” may be used as information for object detection, for example.The “information corresponding to depth” may be in any form ofinformation regarding a “parallax amount (an image shift amount)” of Aimage signals and B image signals generated in the process of conversioninto “distance”, information regarding a “defocus amount”, andinformation regarding “object distance”, for example. In addition, inthis embodiment, the “object distance” among the “informationcorresponding to depth” is acquired in a state of being dispersed overthe entire screen, as information for object detection. Note that the“information corresponding to depth” for object detection may berecorded in association with the image.

The present invention can be applied to various embodiments asinformation corresponding to the depths of the objects in an image.Accordingly, it suffices for information (depth information) indicatedby data corresponding to the depths of the objects to be informationdirectly indicating the object distances in the image from the imagecapturing apparatus to the object or information indicating the relativerelationship between the object distances and the depths of the objectin the image.

Specifically, the image sensor 3 can output images formed as opticalimages by a pair of light beams that pass through different pupilregions of the optical system 1, as paired image signals, from aplurality of photoelectric conversion portions. An image shift amount ofeach region is calculated by correlation calculation between the pairedimage signals, and an image shift map indicating the distribution of theimage shift amounts is calculated. Alternately, the image shift amountis further converted into a defocus amount, and a defocus map indicatingdefocus amount distribution (distribution on the two dimensional planesof the captured image) is generated. If this defocus amount is convertedinto an object distance based on the conditions of the optical system 1or the image sensor 3, distance map data that indicates an objectdistance distribution is acquired.

As described above, in this embodiment, it suffices for the focusdetection unit 6 to acquire image shift map data, defocus map data, ordistance map data of object distances converted from a defocus amount.Note that data of each map data may be held in units of blocks, or inunits of pixels. In this case, about eight bits as the number of bitsare assigned in the smallest unit as in normal image data, and imageprocessing, displaying, recording and the like may be performed usingthe data as distance image, similarly to image processing.

The object tracking unit 7 detects objects based on image signals fromthe image sensor 3 and distance information from the AF control unit 8,specifies a main object from the detected objects, and outputs objectinformation regarding the position and the size of the main object tothe AF control unit 8. If the object tracking unit 7 tracks the face ofa specific person as the main object (main face), a face at a positioncloser to the center of the screen is set as the main face, and thedestination of the main face is detected from the movement vector, colorand size of the main face. The object tracking unit 7 then tracks themain face based on distance information of the main face and distanceinformation of an object around the main face, and determines the mainface in cases such as where another object and the main face pass eachother.

The AF control unit 8 outputs region information to the image sensordriving unit 4 based on the contrast evaluation value from a contrastevaluation value calculation unit 5 and distance information from thefocus detection unit 6.

In addition, in the case of performing contrast AF, the AF control unit8 detects an in-focus position (a peak position at which the contrastevaluation value is largest) based on a contrast evaluation value fromthe contrast evaluation value calculation unit 5, and outputs, to theoptical system driving unit 2, optical system driving information forbringing the main object into the in-focus state. Also, in the case ofperforming phase difference AF, the AF control unit 8 detects anin-focus position based on distance information (corresponding to animage shift amount or a defocus amount at which the correlation amountis smallest) from the focus detection unit 6, and outputs, to theoptical system driving unit 2, optical system driving information forbringing the main object into the in-focus state. Note that the AFcontrol unit 8 may perform control so as to bring the main object closerto the in-focus state (using distance information from the focusdetection unit 6) by performing phase difference AF, and to bring themain object into the in-focus state (using contrast evaluation value) byfurther performing contrast AF. In other words, the AF control unit 8may perform control so as to bring the main object into the in-focusstate using at least one of the contrast evaluation value from thecontrast evaluation value calculation unit 5 and the distanceinformation from the focus detection unit 6.

The signal processing unit 9 generates image data by converting imagesignals from the image sensor 3 into luminance signals and colordifference signals, and outputs the image data to the recording unit 10and the display unit 11. The recording unit 10 and the display unit 11record and display the image data generated by the signal processingunit 9.

Processing for Calculating Optical System Driving Information and RegionInformation in AF Control Unit 8

Next, processing for calculating optical system driving information andregion information performed by the AF control unit 8 will be describedwith reference to FIG. 7.

First, processing for calculating optical system driving information tobe output to the optical system driving unit 2 performed by the AFcontrol unit 8 will be described.

In FIG. 7, a first optical system driving amount calculation unit 700calculates a first optical system driving amount based on a contrastevaluation value from the contrast evaluation value calculation unit 5,by a so-called mountain climbing method. A second optical system drivingamount calculation unit 701 calculates a second optical system drivingamount based on distance information from the focus detection unit 6. Anoptical system driving amount addition unit 702 weights and adds thefirst and second optical system driving amounts based on a contrastevaluation value to be described later, and calculates optical systemdriving information to be output to the optical system driving unit 2.

As described above, contrast AF uses a so-called mountain climbingmethod, and it takes time to calculate a lens driving amount so as toincrease the contrast evaluation value while moving the focus lens, andbring an object into an in-focus state. On the other hand, phasedifference AF makes it possible to calculate a very accurate lensdriving amount even in the case of heavy blur, since distanceinformation to the object can be acquired. Therefore, the lens can bedriven to close to a focus position quickly, but focus adjustmentaccuracy near the focus position is lower compared to contrast AF.

In view of this, in this embodiment, as shown in FIG. 8, the opticalsystem driving unit 2 weights the first optical system driving amountand the second optical system driving amount according to the contrastevaluation value. Specifically, a coefficient α is set such that thelarger the contrast evaluation value is, the larger the weight of thefirst optical system driving amount becomes, and the smaller thecontrast evaluation value is, the smaller the weight of the secondoptical system driving amount becomes, and addition is performed usingthe following expression, for example.

MV=α×Caf+(1−α)×R*Daf

Here, α is the weight coefficient, Caf is the first optical systemdriving amount, and Daf is the second optical system driving amount.

Next, processing for calculating region information to be output to theimage sensor driving unit 4 performed by the AF control unit 8 will bedescribed.

A distance information acquisition region setting unit 703 in FIG. 7sets region information based on a contrast evaluation value that isfrom the contrast evaluation value calculation unit 5 and objectinformation (position and size) of a main object that is from the objecttracking unit 7.

Operation of AF Control Unit

Next, processing for outputting region information to the image sensordriving unit 4 at the time of AF control during a shooting operation,performed by the distance information acquisition region setting unit703 of the AF control unit 8, will be described with reference to FIG.9.

In step S900, the distance information acquisition region setting unit(hereinafter, region setting unit) 703 acquires a contrast evaluationvalue from the contrast evaluation value calculation unit 5.

In step S901, the region setting unit 703 acquires object information(position and size) of a main object from the object tracking unit 7.

In step S902, the region setting unit 703 determines the AF state basedon the contrast evaluation value acquired in step S901. The regionsetting unit 703 uses thresholds Th1 and Th2 (Th1<Th2) to determine thatthe AF state is a heavily blurred state if the contrast evaluation valueis smaller than the threshold Th1, determine that the AF state is amoderately blurred state if the contrast evaluation value is greaterthan the threshold Th1 and smaller than the threshold Th2, and determinethat the AF state is an in-focus state if the contrast evaluation valueis greater than the threshold Th2. The procedure then advances to stepS903 if it is determined that the AF state is the heavily blurred state,the procedure advances to step S904 if it is determined that the AFstate is the moderately blurred state, and the procedure advances tostep S905 if it is determined that the AF state is the in-focus state.

Here, as described above, the weight coefficient α is set such that thelarger the contrast evaluation value is, the greater the weight of thefirst optical system driving amount becomes, and the smaller thecontrast evaluation value is, the smaller the weight of the secondoptical system driving amount becomes. Therefore, when the contrastevaluation value is large, deterioration in AF control accuracy is smalleven if the accuracy in calculation of the second optical system drivingamount is reduced. In opposite terms, if the contrast evaluation valueis small, accuracy in calculation of the second optical system drivingamount must not be reduced.

Moreover, in the object tracking unit 7, a distance map for objectdetection is not necessary if the contrast evaluation value is too smallto determine a main object, and is necessary if the contrast evaluationvalue is large enough to identify the main object. In view of this, inthis embodiment, the frequency of frames in which a distance informationacquisition region for AF control is set and the frequency of frames inwhich distance information acquisition regions for object detection areset are variably controlled according to the contrast evaluation value,thus making it possible to acquire a distance map that can ensure bothAF control accuracy and object tracking accuracy.

Here, control performed so as to switch the frequency at which adistance information acquisition region is set in steps S903 to S905 inFIG. 9 will be described in detail with reference to FIGS. 10A to 10C.Note that in FIGS. 10A to 10C, (a-1) to (a-4), (b-1) to (b-4) and (c-1)to (c-4) respectively indicate frames of captured image signals in timeseries. In addition, in FIGS. 10A to 10C, reference numeral 1000 denotesa main object, reference numeral 1001 denotes an object other than themain object, and reference numeral 1002 denotes a distance informationacquisition region for AF control. Also, reference numeral 1003 denotesa distance information acquisition region for object detection.Moreover, FIG. 10A illustrates an example of region setting in a heavilyblurred state (step S903), FIG. 10B illustrates an example of regionsetting in a moderately blurred state (step S904), and FIG. 10Cillustrates an example of region setting in an in-focus state (stepS905).

In the case of the heavily blurred state in step S903, the regionsetting unit 703 performs region setting such that the frequency offrames in which the distance information acquisition region for AFcontrol 1002 is set is a maximum as in FIG. 10A.

In the case of the moderately blurred state in step S904, the regionsetting unit 703 performs region setting such that the frequency offrames in which the distance information acquisition region for AFcontrol 1002 is set is moderate as in FIG. 10B. In the moderatelyblurred state in FIG. 10B, the frequency of frames in which the distanceinformation acquisition region for AF control 1002 is set lower than inthe heavily blurred state in FIG. 10A, and the frequency of frames inwhich distance information acquisition regions for object detection areset is higher.

The AF state in step S905 is the in-focus state, and thus the regionsetting unit 703 performs region setting so as to reduce the frequencyof frames in which a distance information acquisition region for AFcontrol is set as in FIG. 10C. In the in-focus state in FIG. 10C, thefrequency of frames in which the distance information acquisition regionfor AF control 1002 is set is further reduced compared to the moderatelyblurred state in FIG. 10B, and the frequency of frames in which distanceinformation acquisition regions for object detection are set isincreased. For example, the ratio of frames in which the distanceinformation acquisition region for AF control 1002 is set and the ratioof frames in which the distance information acquisition regions forobject detection are set are the same as in FIG. 10C.

In step S906, the region setting unit 703 calculates region informationaccording to the setting frequency of a distance information acquisitionregion for AF control set in steps S903 to S905, and outputs the regioninformation to the image sensor driving unit 4.

In step S907, the region setting unit 703 determines whether or not theshooting operation has ended, using, as a trigger, an instruction to endthe shooting from the user via the operation unit 12 or the like, andrepeats the processing from step S900 until it is determined that theoperation has ended.

According to this embodiment, the ratio of frames in which a distanceinformation acquisition region for AF control is set and the ratio offrames in which distance information acquisition regions for objectdetection are set are variably controlled according to the contrastevaluation value at the time of AF control during image shooting. Thismakes it possible to ensure both AF control accuracy and main objecttracking accuracy.

Note that in the above embodiment, a distance information acquisitionregion for AF control is set based on object information (position andsize) of a main object, but the region may be set according to an AFmode or an object designated by the user. Moreover, the AF state isdetermined based on the contrast evaluation value, but the AF state maybe determined by the focus detection unit 6 by a phase differencemethod.

Second Embodiment

Next, a second embodiment will be described.

In the first embodiment, the region setting unit 703 controls the ratioof frames in which a distance information acquisition region for AFcontrol is set and the ratio of frames in which distance informationacquisition regions for object detection are set, based on the contrastevaluation value. In contrast, in the second embodiment, a distanceinformation acquisition region setting unit (hereinafter, region settingunit) 1103 in FIG. 11 controls the ratio for a distance informationacquisition region for AF control and the ratio for distance informationacquisition regions for object detection, based on the number of objectsother than a main object.

Note that in the second embodiment, the same reference numerals areassigned to constituent elements similar to those of the firstembodiment, and description thereof is omitted. Differences from thefirst embodiment are that the object tracking unit 7 detects the numberof objects in a captured image based on image signals from the imagesensor 3 and distance information from the focus detection unit 6, andoutputs the detected number of objects to the AF control unit 8, andthat the AF control unit 8 controls the ratio for a distance informationacquisition region for AF control and the ratio for distance informationacquisition regions for object detection, based on the number of objectsfrom the object tracking unit 7.

First, processing for outputting region information to the image sensordriving unit 4 at the time of AF control during a shooting operation,performed by the region setting unit 1103 of the AF control unit 8, willbe described with reference to FIG. 12.

In step S1200, the region setting unit 1103 acquires object information(position and size) of a main object from the object tracking unit 7.

In step S1201, the region setting unit 1103 acquires the number ofobjects other than the main object from the object tracking unit 7.

In step S1202, the region setting unit 1103 determines whether thenumber of objects other than the main object acquired in step S1201 islarge, small or moderate. The region setting unit 1103 uses thresholdsTh3 and Th4 (Th3<Th4) to determine that the number of objects other thanthe main object is small if the number of objects other than the mainobject is smaller than the threshold Th3, determine that the number ofobjects other than the main object is moderate if the number of objectsother than the main object is greater than the threshold Th3 and smallerthan the threshold Th4, and determine that the number of objects otherthan the main object is large if the number of objects other than a mainobject is greater than the threshold Th4. The procedure then advances tostep S1203 if it is determined that the number of objects other than themain object is small, the procedure advances to step S1204 if it isdetermined that the number of objects other than the main object ismoderate, and the procedure advances to step S1205 if it is determinedthat the number of objects other than the main object is large.

In this embodiment, the number of objects other than a main objectserves as a determination condition because the greater the number ofobjects other than the main object is, the higher the possibilitybecomes that an object other than the main object passes in front of themain object, and thus a possibility that determination of the mainobject using distance information is required becomes higher.

In step S1203, if the number of objects other than the main object islarge, the region setting unit 1103 performs region setting so as todecrease the frequency of frames in which the distance informationacquisition region for AF control 1002 is set, and to increase thefrequency of frames in which distance information acquisition regionsfor object detection are set, similar to FIG. 10C.

In step S1204, if the number of objects other than the main object ismoderate, the region setting unit 1103 performs region setting such thatthe frequency of frames in which the distance information acquisitionregion for AF control 1002 is set is higher than in FIG. 10C, and islower than in FIG. 10A, similarly to FIG. 10B.

In step S1205, if the number of objects other than the main object issmall, the region setting unit 1103 performs region setting so as tofurther increase the frequency of frames in which the distanceinformation acquisition region for AF control 1002 is set, similarly toFIG. 10A.

In step S1206, the region setting unit 1103 calculates regioninformation according to the setting frequency of the distanceinformation acquisition region for AF control set in steps S1203 toS1205, and outputs the region information to the image sensor drivingunit 4.

In step S1207, the region setting unit 1103 determines whether or notthe shooting operation has ended, using, as a trigger, an instruction toend the shooting from the user via the operation unit 12 or the like,and repeats the processing from step S1200 until it is determined thatthe operation has ended.

According to this embodiment, the ratio of frames in which a distanceinformation acquisition region for AF control is set and the ratio offrames in which distance information acquisition regions for objectdetection are set are variably controlled according to the number ofobjects other than a main object at the time of AF control during imageshooting. This makes it possible to ensure both AF control accuracy andmain object tracking accuracy.

Note that in the above embodiment, a distance information acquisitionregion for AF control is set based on the number of objects other than amain object, but a configuration may be adopted in which the movementvector of an object other than the main object is detected, and if it isdetermined that a possibility is high that the object other than themain object passes in front of the main object, the ratio for thedistance information acquisition region for AF control is decreased, andif it is determined that a possibility is low that the object other thanthe main object passes in front of the main object, the ratio for thedistance information acquisition region for AF control is increased.

Third Embodiment

Next, a third embodiment will be described.

In the first embodiment, the region setting unit 703 controls the ratioof frames in which a distance information acquisition region for AFcontrol is set and the ratio of frames in which distance informationacquisition regions for object detection are set, based on a contrastevaluation value. In contrast, in the third embodiment, the regionsetting unit 703 sets the priority of a distance information acquisitionregion for AF control in a frame, based on a contrast evaluation value.The region setting unit 703 then controls the ratio for a distanceinformation acquisition region for AF control and the ratio for distanceinformation acquisition regions for object detection that are arrangedin a frame, using ratios that are based on the priority that has beenset.

Note that in the third embodiment, the same reference numerals areassigned to constituent elements similar to those in the firstembodiment, and description thereof is omitted.

Processing for outputting region information to the image sensor drivingunit 4 at the time of AF control during a shooting operation, performedby the region setting unit 703 of the AF control unit 8, will bedescribed below with reference to FIG. 13. Note that steps S1300 toS1302 in FIG. 13 are similar to steps S900 to S902 in FIG. 9, and thusdescription thereof is omitted.

Here, processing for setting a priority for a distance informationacquisition region in steps S1303 to S1305 in FIG. 13 will be describedin detail with reference to FIGS. 14A to 14C.

In FIGS. 14A to 14C, reference numeral 1400 denotes a main object,reference numeral 1401 denotes an object other than the main object, andreference numeral 1402 denotes a distance information acquisition regionfor AF control. Also, reference numeral 1403 denotes a distanceinformation acquisition region for object detection. Moreover, FIG. 14Aillustrates an example of region setting in a heavily blurred state(step S1303), FIG. 14B illustrates an example of region setting in amoderately blurred state (step S1304), and FIG. 14C illustrates anexample of region setting in an in-focus state (step S1305).

In the case of the heavily blurred state in step S1303, the regionsetting unit 703 performs region setting such that the priority of thedistance information acquisition region for AF control 1402 is high.This increases the ratio for the distance information acquisition regionfor AF control 1402 arranged in a frame as in FIG. 14A, and enlarges thedistance information acquisition region for AF control 1402.

In the case of the moderately blurred state in step S1304, the regionsetting unit 703 performs region setting such that the priority of thedistance information acquisition region for AF control 1402 is moderate.In the moderately blurred state in FIG. 14B, the ratio for the distanceinformation acquisition region for AF control 1402 arranged in a frameis smaller compared to the heavily blurred state in FIG. 14A, but thedistance information acquisition regions for object detection 1403 areset so as to be relatively small but discrete over the entire screen.

The AF state in step S1305 is an in-focus state, and thus the regionsetting unit 703 performs region setting so as to lower the priority ofa distance information acquisition region for AF control. In thein-focus state in FIG. 14C, the ratio for the distance informationacquisition region for AF control 1402 in the frame is even smallercompared to the moderately blurred state in FIG. 14B, but the distanceinformation acquisition regions for object detection 1403 are set to berelatively large and discrete over the entire screen.

In step S1306, the region setting unit 703 calculates region informationaccording to the priority of the distance information acquisition regionfor AF control set in steps S1303 to S1305, and outputs the regioninformation to the image sensor driving unit 4.

In step S1307, the region setting unit 703 determines whether or not theshooting operation has ended, using, as a trigger, an instruction to endthe shooting from the user via the operation unit 12 or the like, andrepeats the processing from step S1300 until it is determined that theoperation has ended.

According to this embodiment, the ratio for a distance informationacquisition region for AF control and the ratio for distance informationacquisition regions for object detection that are arranged in a frameare controlled according to the AF state at the time of AF controlduring image shooting. This makes it possible to ensure both AF controlaccuracy and main object tracking accuracy.

Note that in the embodiment above, a distance information acquisitionregion for AF control is set based on object information (position andsize) of a main object, but may be set according to the AF mode or anobject designated by the user. Moreover, the AF state is determinedbased on a contrast evaluation value, but may be determined by the focusdetection unit 6 using a phase difference method.

Fourth Embodiment

Next, a fourth embodiment will be described.

In the second embodiment, the region setting unit 1103 controls theratio for a distance information acquisition region for AF control andthe ratio for distance information acquisition regions for objectdetection, based on the number of objects other than a main object. Incontrast, in the fourth embodiment, a region setting unit 1103 sets thepriority of a distance information acquisition region for AF control ina frame based on the number of objects other than a main object. Theregion setting unit 1103 then controls the ratio for a distanceinformation acquisition region for AF control and the ratio for distanceinformation acquisition regions for object detection that are arrangedin the frame, using ratios that are based on the set priority.

Note that in the fourth embodiment, the same reference numerals areassigned to constituent elements similar to those in the secondembodiment and description thereof is omitted.

Processing for outputting region information to an image sensor drivingunit 4 at the time of AF control during a shooting operation, performedby a region setting unit 1103 of an AF control unit 8, will be describedbelow with reference to FIG. 15. Note that steps S1500 to S1502 in FIG.15 are similar to steps S1200 to S1202 in FIG. 12, and thus descriptionthereof is omitted.

In step S1502, the region setting unit 1103 determines whether thenumber of objects other than a main object acquired in step S1501 islarge, moderate or small. The region setting unit 1103 uses thresholdsTh3 and Th4 (Th3<Th4) to determine that the number of objects other thanthe main object is small if the number of objects other than the mainobject is smaller than the threshold Th3, determine that the number ofobjects other than the main object is moderate if the number of objectsother than the main object is greater than the threshold Th3 and smallerthan the threshold Th4, and determine that the number of objects otherthan the main object is large if the number of objects other than themain object is greater than the threshold Th4. Subsequently, theprocedure advances to step S1503 if it is determined that the number ofobjects other than the main object is small, the procedure advances tostep S1504 if it is determined that the number of objects other than themain object is moderate, and the procedure advances to step S1505 if itis determined that the number of objects other than the main object islarge.

In this embodiment, the number of objects other than a main object isdetermined because the greater the number of objects other than the mainobject is, the more the possibility increases that an object other thanthe main object passes in front of the main object, and thus apossibility that determination of the main object using distanceinformation is required becomes higher.

In step S1503, if the number of objects other than the main object islarge, the region setting unit 1103 performs region setting such thatthe priority of a distance information acquisition region for AF controlbecomes low. Accordingly, similarly to FIG. 14C, a distance informationacquisition region for AF control 1402 in the frame becomes smaller, butdistance information acquisition regions for object detection 1403 areset to be relatively large and discrete over the entire screen.

In step S1504, if the number of objects other than the main object ismoderate, the region setting unit 1103 performs region setting such thatthe priority of the distance information acquisition region for AFcontrol 1402 is moderate. Accordingly, similarly to FIG. 14B, thedistance information acquisition region for AF control 1402 in the framebecomes smaller, but the distance information acquisition regions forobject detection 1403 are set to be relatively small but discrete overthe entire screen.

In step S1505, if the number of objects other than the main object issmall, the region setting unit 1103 performs region setting such thatthe priority of the distance information acquisition region for AFcontrol 1402 is high. This enlarges the distance information acquisitionregion for AF control 1402 in the frame, similarly to FIG. 14A.

In step S1506, the region setting unit 1103 calculates regioninformation according to the priority of setting a distance informationacquisition region for AF control set in steps S1503 to S1505, andoutputs the region information to the image sensor driving unit 4.

In step S1507, the region setting unit 1103 determines whether or notthe shooting operation has ended, using, as a trigger, an instruction toend the shooting from the user via an operation unit 12 or the like, andrepeats the processing from step S1500 until it is determined that theoperation has ended.

According to this embodiment, the ratio for a distance informationacquisition region for AF control and the ratio for distance informationacquisition regions for object detection that are arranged in a frameare controlled according to the number of objects other than a mainobject at the time of AF control during image shooting. This makes itpossible to ensure both AF control accuracy and main object trackingaccuracy.

Note that in the embodiment above, a distance information acquisitionregion for AF control is set based on the number of objects other than amain object, but a configuration may be adopted in which the movementvector of an object other than a main object is detected, and the ratiofor a distance information acquisition region for AF control is loweredif it is determined that a possibility that the object other than themain object passes in front of the main object is high, and the ratio isincreased if it is determined that the possibility is low.

OTHER EMBODIMENTS

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2016-086574, filed Apr. 22, 2016 which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image capturing apparatus comprising: an imagesensor; a readout unit configured to read out a signal from pixels ofthe image sensor; a setting unit configured to set a region in which aplurality of signals having different viewpoints are read out from eachpixel of the image sensor by the readout unit; a first informationacquisition unit configured to acquire first depth information fordetecting an object using a signal that has been read out from a firstregion set by the setting unit; a second information acquisition unitconfigured to acquire second depth information for detecting a focusstate of the object using a signal that has been read out from a secondregion set by the setting unit; and a control unit configured tovariably control a ratio of screens in which the first region is set bythe setting unit and a ratio of screens in which the second region isset.
 2. An image capturing apparatus comprising: an image sensor; areadout unit configured to read out a signal from pixels of the imagesensor; a setting unit configured to set a region in which a pluralityof signals having different viewpoints are read out from each pixel ofthe image sensor by the readout unit; a first information acquisitionunit configured to acquire first depth information for detecting anobject using a signal that has been read out from a first region set bythe setting unit; a second information acquisition unit configured toacquire second depth information for detecting a focus state of theobject using a signal that has been read out from a second region set bythe setting unit; and a control unit configured to variably controlratios of an entire screen occupied by the first region and the secondregion set by the setting unit.
 3. The apparatus according to claim 1,further comprising: an object detection unit configured to detect anobject based on the first depth information, and to detect a specificobject among detected objects; and a focus adjustment unit configured todetermine a focus state of the specific object based on the second depthinformation, and to perform focus adjustment so as to bring the specificobject into an in-focus state.
 4. The apparatus according to claim 3,wherein the first region is a region that is discretely arranged overthe entire screen and is set for object detection performed by theobject detection unit, and the second region is a region that is locallyarranged on the screen and is set for focus adjustment performed by thefocus adjustment unit.
 5. The apparatus according to claim 3, furthercomprising: a third information acquisition unit configured to acquireevaluation information for detecting a focus state of an object using asignal that has been read out by the readout unit, wherein the closerthe evaluation information is to the in-focus state, the lower thesetting unit sets the ratio for the second region, and the farther theevaluation information is from the in-focus state, the higher thesetting unit sets the ratio for the second region.
 6. The apparatusaccording to claim 3, wherein the larger the number of objects otherthan the specific object is, the lower the setting unit sets the ratiofor the second region, and the smaller the number of objects other thanthe specific object is, the higher the setting unit sets the ratio forthe second region.
 7. The apparatus according to claim 5, wherein thecloser the evaluation information is to the in-focus state, the lowerthe setting unit sets a frequency at which the second region is set, andthe farther the evaluation information is from the in-focus state, thehigher the setting unit sets the frequency at which the second region isset.
 8. The apparatus according to claim 6, wherein the larger thenumber of objects other than the specific object is, the lower thesetting unit sets a frequency at which the second region is set, and thesmaller the number of objects other than the specific object is, thehigher the setting unit sets the frequency at which the second region isset.
 9. The apparatus according to claim 5, wherein the closer theevaluation information is to the in-focus state, the lower the settingunit sets a priority of the second region, and the farther theevaluation information is from the in-focus state, the higher thesetting unit sets the priority of the second region.
 10. The apparatusaccording to claim 6, wherein the larger the number of objects otherthan the specific object is, the lower the setting unit sets a priorityof the second region, and the smaller the number of objects other thanthe specific object is, the higher the setting unit sets the priority ofthe second region.
 11. The apparatus according to claim 5, wherein thefirst depth information and the second depth information are informationregarding an object distance acquired by performing correlationcalculation on the signals having different viewpoints, and theevaluation information is information regarding an object contrastacquired from a signal acquired by adding together the signals havingdifferent viewpoints.
 12. The apparatus according to claim 11, whereinthe object detection unit detects information regarding a position and asize of the specific object, and the setting unit sets the first regionand the second region based on the information regarding the positionand the size of the specific object and the information regarding thecontrast.
 13. The apparatus according to claim 6, wherein the objectdetection unit detects information regarding a position and a size ofthe specific object, and the number of objects other than the specificobject, and the setting unit sets ratios for the first region and thesecond region based on the information regarding the position and thesize of the specific object and the number of objects other than thespecific object.
 14. The apparatus according to claim 5, wherein thefocus adjustment unit has a driving amount calculation unit configuredto calculate, based on the second depth information and the evaluationinformation, optical system driving information for driving an opticalsystem so as to bring the specific object into the in-focus state, andthe driving amount calculation unit calculates the optical systemdriving information by weighting a first driving amount calculated fromthe evaluation information and a second driving amount calculated fromthe second depth information.
 15. The apparatus according to claim 14,wherein the closer the evaluation information is to an in-focus state,the higher the driving amount calculation unit sets a weight of thefirst driving amount, and the farther the evaluation information is fromthe in-focus state, the lower the driving amount calculation unit sets aweight of the second driving amount.
 16. The apparatus according toclaim 1, wherein in the image sensor, a plurality of photoelectricconversion portions are assigned to one microlens.
 17. A control methodof an image capturing apparatus which has an image sensor, a readoutunit configured to read out a signal from pixels of the image sensor,and a setting unit configured to set a region in which a plurality ofsignals having different viewpoints are read out from each pixel of theimage sensor by the readout unit, the method comprising: acquiring firstdepth information for detecting an object using a signal that has beenread out from a first region set by the setting unit; acquiring seconddepth information for detecting a focus state of the object using asignal that has been read out from a second region set by the settingunit; and variably controlling a ratio of screens in which the firstregion is set by the setting unit and a ratio of screens in which thesecond region is set.
 18. A control method of an image capturingapparatus which has an image sensor, a readout unit configured to readout a signal from pixels of the image sensor, and a setting unitconfigured to set a region in which a plurality of signals havingdifferent viewpoints are read out from each pixel of the image sensor bythe readout unit, the method comprising: acquiring first depthinformation for detecting an object using a signal that has been readout from a first region set by the setting unit; acquiring second depthinformation for detecting a focus state of the object using a signalthat has been read out from a second region set by the setting unit; andvariably controlling ratios of an entire screen occupied by the firstregion and the second region set by the setting unit.
 19. Acomputer-readable storage medium storing a program for causing acomputer to execute a control method of an image capturing apparatuswhich has an image sensor, a readout unit configured to read out asignal from pixels of the image sensor, and a setting unit configured toset a region in which a plurality of signals having different viewpointsare read out from each pixel of the image sensor by the readout unit,the method comprising: acquiring first depth information for detectingan object using a signal that has been read out from a first region setby the setting unit; acquiring second depth information for detecting afocus state of the object using a signal that has been read out from asecond region set by the setting unit; and variably controlling a ratioof screens in which the first region is set by the setting unit and aratio of screens in which the second region is set.
 20. Acomputer-readable storage medium storing a program for causing acomputer to execute a control method of an image capturing apparatuswhich has an image sensor, a readout unit configured to read out asignal from pixels of the image sensor, and a setting unit configured toset a region in which a plurality of signals having different viewpointsare read out from each pixel of the image sensor by the readout unit,the method comprising: acquiring first depth information for detectingan object using a signal that has been read out from a first region setby the setting unit; acquiring second depth information for detecting afocus state of the object using a signal that has been read out from asecond region set by the setting unit; and variably controlling ratiosof an entire screen occupied by the first region and the second regionset by the setting unit.